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PREFACE 


This  document  was  prepared  under  a  task  titled  “Critical  Technology  Issues  for 
the  Future  Combat  Systems  Command  and  Control”  for  the  Tactical  Technology  Office, 
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succeeded  by  Franklin  L.  Moses  at  the  Institute  for  Defense  Analyses. 
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EXECUTIVE  SUMMARY 


BACKGROUND 

Assessments  of  small  air  and  ground  robot  systems  often  focus  on  how  well  the 
equipment  functions.  Those  assessments  also  should  include  the  human  operator  and  how 
well  the  robot  and  operator  work  together  as  an  integrated  system.  Tests  that  do  include 
performance  of  the  human  operator  often  rely  on  qualitative  observations — observer 
judgments  and  interviews  about  workload,  situation  awareness,  cognitive  issues,  and  so 
on.  This  paper  views  the  operator  and  robot  as  a  team,  outlines  a  schema  for  measuring 
robot-operator  team  performance,  and  presents  an  initial  proof-of-principle  test  for 
quantitatively  assessing  that  performance. 

The  initial  proof-of-principle  test  (1)  defined  robot-operator  performance  factors 
associated  with  moving  a  small  robot  from  point  A  to  point  B  and  (2)  quantified  the 
effects  that  different  sensor  and  navigation  technologies  have  on  that  performance. 

METHODOLOGY 

The  proof-of-principle  approach  was  to  instrument  the  robot  and  the  operator 
interface  to  allow  measures  of  operational  performance  in  navigation-reconnaissance 
tasks.  The  instrumentation  enabled  measurement  of  efficiency  and  effectiveness  (e.g., 
frequency  of  control  actions,  time  between  control  actions)  and  errors  and  accuracy. 
These  kinds  of  measurements  provide  data  about  mission  performance  contributions 
spanning  robot  capabilities,  operator  skills,  employment  strategy,  interface  limitations, 
and  so  on. 

System  Components 

A  team  consisted  of  a  single  operator  and  robot.  An  experimenter/observer  ran  the 
tests.  The  robot  was  equipped  for  operator  control  using  teleoperation  via  Ethernet  or  for 
operator  control  with  the  help  of  electronic  navigation  aids. 

The  operator’s  workstation  had  a  laptop  to  provide  a  control  display  with  four 
information  areas:  (1)  map,  including  robot  position  and  activity;  (2)  error  feedback 
window;  (3)  live-camera  video,  and  (4)  quick-reference  command  list  showing  all 
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operator-robot  inputs.  The  map  on  the  laptop’s  display  showed  the  test  course  of 
approximately  135  x  95  feet  in  a  level-floor  parking  garage.  The  test  course  consisted  of 
gates  and  obstacles  for  the  robot-operator  team  to  navigate  through  and  around.  Operator 
control  of  the  robot  was  done  using  the  laptop’s  keyboard  and  mouse. 

Procedure 

An  observer  showed  and  explained  the  robot  and  its  features  to  the  operator.  The 
observer  also  showed  and  explained  a  training  course  and  its  features  but  not  the  test 
course. 

There  were  two  different  navigation  conditions:  unaided  and  aided.  Without 
electronic  navigation  (unaided),  the  operator  perfonned  all  control  via  teleoperation,  the 
most  common  way  to  operate  robots  today.  With  electronic  navigation  and  teamwork 
(aided),  the  robot  assisted  the  operator  in  navigating  a  route  in  two  ways:  (1)  it  followed  a 
sequence  of  operator-selected  waypoints,  and  (2)  it  avoided  3D  objects.  The  navigation 
task  for  both  conditions  was  to  drive  the  robot  through  a  series  of  numbered  gates  while 
avoiding  obstacles. 

RESULTS  AND  DISCUSSION 

Analyses  compared  individual  operator  performance  under  aided  and  unaided 
conditions  for  each  test  run.  Control  actions — the  number  of  key  presses  as  well  as  the 
mean  times  and  ranges  of  times  between  key  presses — were  calculated.  Error  frequency 
also  was  compiled.  Table  ES-1  gives  the  results  of  this  comparison.  Compared  with 
unaided  navigation,  aided  navigation  had  fewer  control  actions,  more  free  time  (intervals 
of  greater  than  5  seconds  between  control  actions),  but  a  higher  number  of  errors. 

Table  ES-1.  Mean  Number  and  Mean  Time  for  Response  Parameters 


Number  of 

Free  Time  / 

Number  of 

Navigation 

Key  Presses 

Navigation 

Errors 

Aided 

12 

7.8  s 

>1 

Unaided 

50 

1.6  s 

<1 

Less  operator  attention  and  interaction  with  the  robot  generally  seems  to  equate 
with  more  errors,  suggesting  that  operator  vigilance  needs  emphasis  or  that  the  navigation 
technology  needs  enhancement  for  improved  performance. 
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CONCLUSION 


The  test  demonstrated  the  usefulness  of  evaluating  robot-operator  teams  to  assess 
success  and  failure  factors  in  the  performance  of  robots  and  the  effects  of  different 
technologies  on  that  performance. 

RECOMMENDATIONS 

The  measurement  concept  developed  and  tested  as  a  proof  of  principle  would 
benefit  from  additional  assessment  and  enhancements  before  transition  to  the  user 
community.  The  next  steps  should  be  to  extend  the  measurement  approach  to  more 
complex  robot-operator  tasks  with  the  goal  of  providing  quantified  answers  to  which 
technologies  will  help  robot-operator  teams  better  accomplish  their  jobs.  The  current 
work  with  one  type  of  robot  and  limited  data  needs  expansion  to  show  that  the  concept 
generalizes.  Additional  tests  should  include  data  logs  and  metrics  for  other  ground  robots 
and  for  air  robots.  To  make  implementation  easy,  the  schema  and  instrumentation  for 
data  collection  need  to  be  relatively  compatible  with  different  onboard  computers,  data¬ 
logging  systems,  and  user  interfaces  and  control  stations  of  various  robots.  The  most 
effective  way  for  that  to  happen  is  for  contractors  and  developers  to  integrate  the 
evaluation  methodology  into  their  robot  systems  and  planning  for  employment  under 
more  realistic  conditions  than  in  the  current  study. 
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I.  INTRODUCTION  AND  PURPOSE 


The  effectiveness  of  a  system  generally  is  determined  by  how  well  it  performs  a 
mission  or  task.  To  improve  a  system,  developers  must  examine  individual  components 
and  assess  their  contribution  to  overall  system  perfonnance.  Assessments  of  small  air  and 
ground  robot  subsystems  often  focus  on  how  well  the  equipment  functions.  However, 
those  assessments  should  also  consider  the  human  operator  subsystem  and  how  well  the 
operator  and  robot  components  work  together  as  an  integrated  system. 

An  Army  Science  Board  assessment  of  robot-operator  interface  issues  concluded 
that  robotic  associated  technologies  themselves  are  not  the  major  issues  in  robot  system 
performance.  Instead,  frequent  shortfalls  include: 

•  Robot-operator  interfaces  are  ad  hoc,  developed  primarily  by  engineers  for 
engineers,  and  not  systematically  evaluated. 

•  No  rigorous  efforts  exist  to  understand  robot  mission  functions,  robot 
limitations,  and  the  consequent  operator  interactions. 

•  No  metrics  exist  for  systematically  improving  operator-robot  interactions. 

In  addition,  tests  that  do  include  performance  of  the  human  operator  often  rely  on 
qualitative  observations — observer  judgments  and  interviews  about  workload,  situation 
awareness,  cognitive  issues,  and  so  on.  They  lack  quantitative  measures  related  to  total 
system  performance.  To  emphasize  the  total  system,  this  paper  views  the  operator  and 
robot  as  a  team.  It  outlines  a  performance  schema  for  that  team,  presents  an  initial  proof- 
of-principle  test  for  quantitatively  assessing  the  team’s  performance,  and  interprets  data 
from  the  test. 

The  initial  proof-of-principle  test  (1)  defines  (robot-operator)  perfonnance  factors 
associated  with  moving  small  robots  from  point  A  to  B  and  (2)  quantifies  the  effects  that 
different  sensor  and  navigation  technologies  have  on  that  performance.  Measures  were 
designed  to  be  as  non-task  and  non-platfonn  specific  as  possible  so  that  generalizations 
could  be  made. 

The  proof-of-principle  approach  was  to  instrument  the  robot  and  the  operator 
interface  to  allow  measures  of  operational  performance  in  navigation-reconnaissance 
tasks.  The  instrumentation  enabled  measurement  of  efficiency  and  effectiveness  (e.g., 
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frequency  of  control  actions,  time  between  control  actions)  and  errors  and  accuracy. 
These  kinds  of  measurements  provided  data  about  mission  performance  contributions 
spanning  robot  capabilities,  operator  skills,  employment  strategy,  interface  limitations, 
and  so  on.  The  payoff  is  that  program  managers  and  developers  can  use  measurement 
schema  like  the  one  described  to  help  diagnose  and  improve  robot-operator  team 
configurations  and  to  assist  in  technology  trade-off  and  investment  decisions. 
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II.  BACKGROUND 


A.  ROBOT-OPERATOR  TEAMWORK 

This  project  approached  the  development  of  measures  from  the  perspective  of 
robot-operator  teams  and  the  effectiveness  and  efficiency  of  their  teamwork.  In  one  of  the 
most  widely  cited  working  definitions  of  a  team,  Salas  et  al.  (1992)  characterized  a  team 
as  having  a  function,  goal,  and  direction: 

...a  distinguishable  set  of  two  or  more  people  who  interact,  dynamically, 
interdependently,  and  adaptively  toward  a  common  and  valued 
goal/objective/mission,  who  have  been  assigned  specific  roles  or  functions 
to  perfonn,  and  who  have  a  limited  life-span  of  membership  [p.  4]. 

This  definition  suggests  that  each  team  member  has  a  specific  role  representing  a  critical 
contribution  to  effective  system  performance.  In  the  robot-operator  team,  each  partner 
has  distinct  tasks  and  unique  responsibilities:  the  operator  plans,  directs,  and  monitors  the 
robot,  while  the  robot  uses  its  sensory  and  surveillance  capabilities  to  investigate 
environments  separate  from  the  operator.  The  robot  and  the  operator  each  have 
responsibility  for  different  aspects  of  perfonnance,  but  they  are  dependent  on  one  another 
to  complete  missions.  Measurement,  therefore,  must  account  for  how  effectively  and 
efficiently  they  each  perform  independent  tasks  and  share  dependent  requirements.  The 
load  on  the  operator  during  an  operation  varies  according  to:  (1)  the  number  of 
interdependent  and  entirely  operator-directed  actions  required  and  (2)  the  features  of  the 
robot  that  supplement  or  take  over  some  operator  functions.  One  day,  robots  may  be 
designed  to  be  more  acutely  sensitive  than  operators  to  environmental  cues  that  can  have 
critical  infonnative  impact  on  operator  decision-making. 

Interdependence  represents  a  key  parameter  of  teams.  Members  coordinate  their 
activities,  either  sequentially  or  simultaneously.  In  robot-operator  teams,  the  mission 
typically  is  initiated  by  the  operator;  however,  during  the  mission,  the  actions  and 
reactions  of  the  operator  become  dependent  upon  actions  and  infonnation  from  the  robot. 
Thus,  robot-operator  activity  reflects  a  high  level  of  interdependence. 
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Further,  the  robot-operator  team  is  designed  for  particular  types  of  missions, 
typically  involving  the  robot’s  entry  into  and  operation  within  dangerous  environments, 
as  directed  by  the  operator. 

The  definition  and  designation  of  the  robot-operator  system  as  a  team  sets  the 
foundation  for  a  measurement  system.  Most  team-performance  models  define 
effectiveness  as  a  function  of  team  processes  or  of  the  interactions  among  team  members. 
Likewise,  Bran  nick  and  Prince  (1997)  argued,  “a  comprehensive  measure  of  team 
performance  needs  to  contain  elements  of  both  process  and  outcome”  (p.  10). 
Accordingly,  performance  measures  that  can  apply  to  robot-operator  teams  reflect  both 
the  electronic -based  interactions  between  operator  and  robot  (e.g.,  operator  keystrokes) 
and  indices  of  mission  outcomes  (e.g.,  mission  accomplishment;  time  to  completion,  and 
error  rates).  A  central  task  in  this  effort  was  to  specify  particular  processes  for  the  robot- 
operator  team  environment  and  to  define  behavior-based  indicators  of  these  processes. 

B.  TEAMWORK  ASSESSMENT 

Two  types  of  functions,  planning  and  coordination,  abstracted  from  Marks  et  al. 
(2001),  are  particularly  critical  for  effective  team  performance.  Planning  occurs  before 
and  in  anticipation  of  team  action,  while  coordination  is  essential  to  actions  for  carrying 
out  the  plans.  Operators  make  the  higher  order  planning  decisions,  while  robots  provide 
information  crucial  to  overall  task  planning,  as  well  as  to  the  adaptation  and  recalibration 
of  plans  during  team  actions. 

Coordination  functions  include  the  simultaneous  synchronization  of  actions  by  the 
robot  and  the  operator.  These  functions  require  feedback  and  monitoring  of  team 
activities,  error  detection  and  correction,  and  backup  behaviors,  where  one  team  member 
can  provide  corrective  information  to  another  and  assist  in  perfonnance.  Coordination 
represents  an  essential  component  of  processes  in  robot-operator  teams. 

The  assessment  of  robot-operator  team  performance  necessarily  rests  on 

(1)  Behavior-based  indices  of  processes  (e.g.,  latency  of  responses  to 
encountered  obstacles) 

(2)  Markers  of  electronic  communications  (e.g.,  frequency  and  pattern  of 
operator  signals  to  the  robot,  frequency  and  pattern  of  robot  signals  to  the 
operator) 

(3)  Outcomes  of  collaborative  activity  (e.g.,  mission  accomplishment,  time  to 
mission  accomplishment,  aggregated  accuracy  and  error  rates) 
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Table  1  lists  indicators  of  team  processes  for  robot-operator  performance.  This  table  also 
provides  examples  of  how  these  processes  can  be  measured  for  the  navigation- 
reconnaissance  tasks  used  in  the  current  project. 


Table  1.  Robot-Operator  Team  Performance  Measures 


Robot-Operator  Team 

Processes  and  Performance 

Measurement  Indicators 

Information  exchanges  about  task  situation 

Frequency  and  timing  of  electronic  exchanges 
(e.g.,  operator  control  actions  anticipating 
obstacles  or  robot  alerts  about  environmental 
obstacles  and  events) 

Directive  information  (planning-into-action) 

Operator  control  actions  and  patterns  (e.g., 
time  actually  spent  controlling  the  robot) 

Monitoring  and  feedback  regarding  goal-path 

Alerts  and  warnings  by  the  robot  to  the 

adherence 

operator  and  frequency  of  queries  by  the 
operator  to  the  robot 

Activity  monitoring  and  backup  behaviors 

Frequency  of  operator  interventions  to  correct 
robot  course  after  encounters  with  obstacles 
and  blockages 

Frequency  of  course  adjustments  made  by 
operator 

Frequency  of  robot  alerts  following  operator 
keystrokes 

Coordinated  (sequential,  simultaneous, 

Aggregated  latency  between  keystrokes  and 

integrated)  actions  by  operator  and  robot 

robot  signals 

Latency  of  operator  corrective  responses  to 
robot 

Frequency  of  robot  “freezes”  (i.e.,  robot 
backed  into  part  of  maze  and  cannot  move 
out) 

Pacing  of  team  activity 

Time  to  sub-goal  completion 

Frequency  and  timing  of  electronic  exchanges 
throughout  mission 

Team  performance 

Mission  accomplishment 

Time  to  mission  accomplishment 

Percentage  of  task  requirements  met 

Aggregated  accuracy  and  error  rates  (errors 
include  frequencies  of  2D  obstacles  crossed 
by  robot,  3D  obstacles  hit  by  robot, 
misidentification  of  environmental  threats  and 
events) 
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The  robot-operator  team  performance  measures  developed  in  this  effort  were 
selected  for  their  sensitivity  to  the  cognitive  load  requirements  on  the  operator.  The 
operator  depends  on  teamwork  with  the  robot  to  reduce  the  amount  of  cognitive  effort 
needed  to  control  it.  The  most  efficient  and  effective  robot-operator  team  performance 
occurs  when  the  operator  can  minimize  both  cognitive  resource  allocation  to,  and 
intervention  in,  robot  activities  once  a  team  mission  and  plan  is  set  in  motion.  In  such 
instances,  the  operator’s  cognitive  resources  can  be  directed  toward  broader  strategic  and 
team-management  issues. 

The  proof-of-principle  test  described  in  the  next  section  of  this  report  was 
designed  to  assess  the  impact  of  robot  technologies  on  teamwork  perfonnance.  The 
implications  for  cognitive  load  reduction  of  different  robot  technologies  vary.  This  study 
applies  the  robot-operator  team  performance  measures  to  assess  the  impact  of  one 
technology  feature,  robot-aided  navigation,  which  can  have  significant  effects  (both 
positive  and  negative)  on  operator  cognitive  load.  This  feature,  while  not  widely  used 
effectively  in  most  applied  robot  settings,  allows  for  a  valid  test  of  the  proposed  measures 
in  a  controlled  environment.  The  validation  of  the  measures  and  their  underlying 
principles  in  this  environment  provides  the  foundation  for  tests  of  their  applicability  in 
less  controlled  environments. 
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III.  METHODOLOGY 


The  purpose  of  this  research  was  to  test  and  measure  how  well  a  robot-operator 
team  performs  under  different  conditions.  The  robot-operator  team  perfonned  navigation 
and  reconnaissance  tasks.  Components  of  the  test,  described  below,  were  the  robot- 
operator  team,  the  operator  workstation,  and  the  test  course. 

A.  SYSTEM  COMPONENTS 

1.  Robot  Operators  and  Experimenters/Observers 

Twelve  people  were  involved  in  testing:  a  total  of  ten  operators  who  were  familiar 
with  computers  but  who  had  no  prior  experience  with  remote  robot  operation,  along  with 
an  experimenter  and  a  backup  to  introduce  the  testing,  monitor  equipment,  and  observe 
performance  for  problems. 

2.  Robot-Operator  Team 

A  team  consisted  of  a  single  operator  and  robot.  The  robot  was  equipped  for 
operator  control  using  teleoperation  via  Ethernet  or  for  operator  control  with  the  help  of 
electronic  navigation  aids.  Figure  1  shows  the  wheeled  robot  (Pioneer  P3-AT  Robot®  by 
ActivMedia  Robotics).1  In  teleoperation  mode,  it  had  a  tilt-and-zoom  color  video  camera, 
laser  and  sonar,  and  bumpers  with  collision-stop  sensors.  For  navigation  with  electronic 
aids,  the  robot  had  two  added  features:  (1)  automated  obstacle  avoidance  that  allows  the 
robot  to  sense  and  avoid  3D  obstacles,  and  (2)  waypoint  navigation  that  allows  the 
operator  to  mark  a  series  of  map  coordinates  for  the  robot  to  follow.  The  robot  also  had 
hot-swappable  batteries  for  power  and  embedded  microprocessors  running  Linux  control 
software  to  manage  the  robot,  data-logging,  and  feedback  to  the  operator.  Ten 
experienced  operators  each  used  a  PC  laptop,  mouse,  and  map  of  the  course  for  robot 
interactions. 


1  The  use  of  registered  trademarks,  companies,  and  brand  names  is  for  accurate  descriptive  purposes 
only  and  does  not  represent  endorsement  of  the  product  by  IDA  or  its  sponsors. 
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—  Color  Video  Camera 


Laser  Rangefinder 


Ethernet  Antenna 
Sonar  Arrays 


Bumpers/Collision-Stop 

Sensors 


Figure  1.  Pioneer  P3-AT  Robot® 


Figure  2  shows  the  relationship  between  the  software  and  hardware  components 
used  to  control  the  robot.  The  functions  of  the  major  components  are: 

•  Laptop  PC — Operator’s  workstation  to  control  the  robot  and  receive  sensor 
feedback. 

•  Color  Video  Camera — Provides  forward-looking  video  from  the  robot  to  the 
laptop  PC. 

•  Video  Frame  Grabber — Samples  the  color  camera’s  image  every  1/5  second 
for  display  on  the  laptop  PC. 

•  Laser  Rangefinder — Robot  navigation  and  mapping  capabilities  with  a  range 
of  10-50  meters  and  readings  every  1  degree  in  a  180-degree  forward- 
looking  arc. 

•  ActivMedia  Robotic  Operating  System  Microcontroller — Controls  all  low- 
level  robot  systems,  including  motor  operation,  firing  the  sonar,  collecting 
sonar  data,  and  collecting  wheel  encoder  data. 

•  Saphira — An  open-source  software  package  developed  by  SRI  that  provides 
semi-autonomous  navigation  control  with  object-collision  avoidance. 

•  ActivMedia  Robotics  Interface  for  Applications — Interface  for  intelligent 
robotics  systems  such  as  Saphira. 
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•  Laptop  PC  Display — Information  needed  by  the  operator  for  robot  control 
that  is  recordable  for  later  replay. 

•  Log  Files — Software  capture  of  all  robot  state  parameters  and  robot  control 
inputs. 

3.  Operator  Work  Station  and  Test  Course 

The  workstation  consisted  of  a  Model  5100  Inspiron  Dell  laptop  with  Windows 
XP  connected  to  a  Linksys-B  Ethernet  router.  The  laptop  provided  a  control  display  with 
four  infonnation  areas  (Figure  3):  (1)  map,  including  robot  position  and  activity;  (2)  error 
feedback  window;  (3)  live-camera  video;  and  (4)  quick-reference  command  list  showing 
all  operator-robot  inputs. 

The  map  on  the  laptop’s  display  showed  the  test  course  represented 
diagrammatically,  but  not  to  scale,  as  in  Figure  4.  This  course  was  approximately  135  x 
95  feet  in  a  level-floor  parking  garage.  The  test  course  consisted  of  gates  and  obstacles. 
Pairs  of  1-1/2  x  2  foot  cardboard  boxes  formed  3D  gates  labeled  with  Roman  numerals. 
Other  free-standing  boxes  provided  additional  3D  obstacles.  2D  obstacles  were  made  by 
yellow-striped  warning  tape  formed  into  2-foot  squares  on  the  floor.  Three  pairs  of  paths 
up  and  back  through  gates  were  defined  on  the  course.  Each  course  had  easier  paths  with 
fewer  steep  angles  and  fewer  obstacles  from  one  gate  to  the  next  and  harder  paths  with 
steeper  angles  and  more  obstacles. 

An  operator  controlled  the  robot  using  the  laptop’s  keyboard  and  mouse.  The 
arrow  keys  and  space  bar  were  for  movement:  go  forward-backward  (up-down  arrows), 
turn  left-right  (left-right  arrows),  and  space  bar  for  stop.  Camera  control  similarly  used 
four  keys:  zoom  in-out  (D  and  A  keys)  and  tilt  up-down  (W  and  S  keys),  with  panning 
controlled  by  turning  the  robot.  The  mouse  was  used  to  start  the  robot’s  motors  and  to 
enter  and  control  a  semi-autonomous  (aided)  navigation  mode  where  the  operator 
established  waypoints  from  one  location  to  another  for  the  robot  to  follow.  A  waypoint 
was  deleted  using  the  backspace  key,  and  a  waypoint  inserted  using  the  mouse  and  Alt 
key. 

The  well-lit  work  station  area  was  about  12  feet  x  12  feet.  It  held  a  second 
computer  work  station  that  shadowed  and  recorded  the  operator’s  display  along  with 
chairs  for  the  operator  and  two  observers.  The  trials  were  run  in  late  fall  with 
temperatures  between  40-50  °F.  Partitions  made  a  cubicle  around  the  workstation  area 
that  blocked  the  operator’s  view  of  the  robot  except  during  training  when  one  side  of  the 
enclosure  was  moved. 
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Figure  2.  Schematic  of  Robot  Control  Elements 
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Figure  3.  Laptop  Control  Display 
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Figure  4.  Schematic  (not  to  scale)  of  Test  Course 


B.  PROCEDURE 


1.  Mapping  the  Course 

The  robot’s  laser  rangefinder  was  used  to  create  a  map  of  the  course,  including 
the  locations  of  the  gates  and  obstacles.  Boxes  were  temporarily  placed  on  2D  obstacles 
to  make  them  detectable  by  the  laser.  An  observer  manually  guided  the  robot  up  and  back 
through  the  course  while  the  robot’s  laser  rangefinder  scanned  and  mapped  all  obstacles. 
This  complete  map  was  later  used  for  scoring  performance.  For  the  tests,  however,  all  2D 
obstacles  were  deleted  from  the  map  and  so  were  some  3D  obstacles.  Thus,  the  operator 
(but  not  the  robot)  could  see  2D  obstacles  with  the  video  camera,  and  the  robot  could 
sense  unmapped  3D  obstacles  with  its  laser  and  sonar,  either  to  alert  the  operator  or  to 
avoid  them.  These  mapping  conditions  helped  distinguish  robot-aided  navigation  from 
unaided  navigation  modes  described  later  in  this  section.  These  conditions  also  mimic  a 
real-world  situation  where,  through  teamwork,  the  operator  and  the  robot  each  detect 
classes  of  obstacles  and  share  infonnation  for  best  control. 

2.  Training 

Before  an  operator  arrived,  the  observer  set  up  the  course  and  workstation  area, 
initialized  the  Ethernet  and  robot,  and  set  the  laptop’s  display  to  show  the  training  area 
map.  The  robot  was  placed  at  a  pre-established  start  point.  An  observer  then  showed  and 
explained  the  robot  and  its  features  to  the  operator.  The  observer  also  showed  and 
explained  the  training  course  and  its  features,  but  not  the  test  course.  Next,  the  operator 
was  given  a  hard-copy  map  of  the  training  course  for  reference,  along  with  self-paced 
instructions  (see  Appendix  A)  for  learning  to  be  facile  controlling  the  robot  before 
starting  actual  tests.  During  training,  the  operator  could  see  the  training  course  and  robot 
behavior  but  was  encouraged  to  practice  also  using  the  computer  display  alone — the  only 
information  available  during  tests. 

3.  Pre-Testing 

Before  testing,  an  observer  changed  the  operator’s  workstation  map  to  the  test 
course,  changed  the  robot’s  batteries,  placed  it  at  the  testing  start  point,  and  began 
recording  the  operator’s  display.  There  were  two  different  navigation  conditions — 
unaided  and  aided. 
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4.  Unaided  Navigation 

Without  electronic  navigation,  the  operator  performed  all  control  via 
teleoperation,  the  most  common  way  to  operate  robots  today.  The  operator  relied  on  a 
hard-copy  map,  an  electronic  map  showing  3D  objects  (except  hidden  ones)  and  the 
robot’s  location,  video  images  of  the  course,  and  feedback  about  robot  movement.  In 
addition,  the  laser  and  sonar  traced  all  3D  objects  within  range  on  the  electronic  map. 

5.  Aided  Navigation 

With  electronic  navigation  and  teamwork,  the  robot  assisted  the  operator  in 
navigating  a  route  in  two  ways:  (1)  it  followed  a  sequence  of  operator-selected 
waypoints,  and  (2)  it  avoided  3D  objects.  The  robot  could  not  detect  2D  obstacles, 
however,  so  the  operator  had  to  use  the  live  video  camera  display  to  see  and  avoid  them. 
During  aided  navigation,  the  operator  could  still  go  into  unaided  navigation  mode  for 
situations  where  the  robot  got  stuck  or  otherwise  needed  help. 

6.  Navigation  Task 

An  operator  steered  the  robot  through  a  series  of  numbered  gates,  either  aided 
with  waypoint  navigation  or  unaided  using  the  robot’s  camera  and  sensors.  The  course 
had  several  possible  numbered  paths  marked  by  green  circles  and  Roman  numerals.  For 
each  test,  operator-robot  teams  went  between  gates  (pairs  of  numbered  boxes)  in  a 
specific  order  (see  Appendix  B)  to  complete  the  course. 

Operators  each  navigated  the  course  six  times.  The  first  two  paths  through  the  test 
course  were  mostly  identical  and  familiarized  operators  with  navigating  the  course,  once 
aided  and  once  unaided.  Four  subsequent  navigation  paths,  two  up  and  two  back,  were  all 
different.  The  operator  navigated  these  paths  twice  with  aided  and  twice  with  unaided 
navigation  using  the  sequence  of  unaided,  aided,  aided,  and  unaided.  Appendix  B  shows 
details  of  the  test  paths,  conditions,  and  tasks. 

An  operator  was  told  to  move  as  fast  as  possible  while  avoiding  3D  and  2D 
obstacles.  At  the  end  of  a  path  up  the  course,  operators  were  told  to  reverse  direction  by 
turning  the  robot  around  two  boxes  stacked  on  top  of  one  another  and  following  the 
observer’s  instructions  about  changing  navigation  condition  (aided  or  unaided)  and  task. 
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7.  Measurement 


Two  categories  of  measures  were  derived  from  the  data:  efficiency  and  errors. 
Efficiency  of  navigation  was  measured  by  (1)  the  frequency  of  control  actions  (number  of 
key  presses)  per  unit  time  and  (2)  the  length  of  time  between  consecutive  control  actions. 
Navigation  errors  or  accuracy  was  measured  by  counting  the  number  of  2D  obstacles  an 
operator  crossed  or  ran  over.  Error  data  from  bumping  3D  objects  was  not  counted 
because  in  aided  mode,  the  robot  could  never  bump  a  3D  object.  Thus,  unaided 
performance  was,  at  best,  no  better  than  aided  perfonnance.  Control  action  measures 
were  obtained  from  data  logs  of  each  type  of  key  press  and  mouse  click  during  the  time  it 
took  to  navigate  the  course.  Errors  were  tallied  manually  by  watching  recordings  of  the 
robot’s  path  through  the  course. 
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IV.  RESULTS  AND  DISCUSSION 


A  test  of  aided  vs.  unaided  navigation  of  robots  provided  support  for  the  argument 
that  electronic  quantification  of  operator-robot  team  performance  is  practical.  The  results 
demonstrated  the  utility  of  a  measurement  system  grounded  in  principles  of  teamwork, 
where  operators  and  robots  have  defined  roles,  and  perfonnance  derives  from  their 
integrated  actions.  Using  team-based  measurement  indices,  the  findings  showed  how 
changes  in  robot  technologies  can  affect  system  perfonnance. 

Data  were  compiled  for  aided  and  for  unaided  conditions  across  the  navigation 
tests  that  each  operator  perfonned.  These  were  summarized  by  calculating  the  arithmetic 
means  (averages)  and  ranges  across  all  operators  for  control  actions — number  of  key 
presses  per  unit  time  as  well  as  the  times  and  ranges  of  times  between  key  presses.  Error 
frequency  also  was  compiled  and  means  calculated  for  the  number  of  obstacles  operators 
failed  to  avoid.  Tests  of  significance  were  not  done  due  to  the  small  number  of  operators 
and  data. 

Overall  results  showed  that  compared  with  unaided  navigation,  aided  navigation 
had  fewer  control  actions,  more  free  time  (intervals  of  greater  than  5  seconds  between 
control  actions),  but  a  higher  number  of  errors.  Table  2  gives  these  three  measures  across 
the  two  conditions  of  aided  and  unaided  perfonnance.  Although  the  metrics  are  different, 
the  comparisons  of  results  grouped  by  condition  are  instructive. 


Table  2.  Mean  Number  and  Mean  Time  for  Response  Parameters 


Number  of 

Free  Time  / 

Number  of 

Navigation 

Key  Presses 

Navigation 

Errors 

Aided 

12 

7.8  s 

>1 

Unaided 

50 

1.6  s 

<1 

The  mean  number  of  key  presses  per  unit  time  for  aided  vs.  unaided  navigation 
was  12  vs.  50.  In  addition,  mean  (free)  time  between  key  presses  was  7.8  seconds  for 
aided  navigation,  ranging  from  1.8  to  20.7  seconds;  for  unaided  navigation,  mean  time 
between  key  presses  was  only  1.6  seconds,  ranging  from  0.5  to  3.2  seconds.  The  mean 
number  of  errors  for  aided  navigation  was  greater  than  one  while  unaided  navigation 
resulted  in  less  than  one  error  per  run. 
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The  time  intervals  between  key  presses  were  further  analyzed  by  summing  them 
by  length:  all  0.1-second  intervals  plus  all  0.2-second  intervals,  and  so  on.  The 
accumulated  percent  number  of  key  press  intervals  (y  axis)  was  then  plotted  as  a  function 
of  the  time  interval  (x  axis)  an  operator  paused  between  key  presses.  Figure  5  shows  both 
aided  and  unaided  navigation  data  for  four  representative  operators. 


Operator  A 


Operator  C 


♦  Aided 
■  Unaided 


Operator  B 


♦  Aided 
■  Unaided 


Operator  D 


♦  Aided 
■  Unaided 


Figure  5.  Key  Presses  as  a  Function  of  Time  Intervals  Between  Key  Presses 

A  striking  characteristic  of  the  data  for  time  between  key  presses  is  the  shapes  of 
the  distributions.  For  the  unaided  mode,  90%  of  those  data  are  less  than  5  seconds.  This 
was  the  case  regardless  of  a  test  run’s  length,  which  varied  considerably  among 
operators.  For  aided  navigation,  on  the  other  hand,  only  10%  to  80%  of  time  interval 
distributions  between  key  presses  were  less  than  5  seconds. 

These  plots  show  that  operator-robot  teams  using  the  aided  navigation  mode  can 
exercise  more  intermittent  control  with  fewer  key  presses  and  more  time  between  them 
than  in  unaided  navigation.  This  means  they  have  more  “free  time”  to  consider  navigation 
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strategy  and  route  planning.  In  contrast,  in  the  unaided  navigation  mode,  operators  had 
less  time  between  inputs  for  these  functions  due  to  a  nearly  continuous  requirement  to 
monitor  and  react  to  robot  feedback. 

Perhaps  the  most  interesting  result  from  these  plots  of  time  intervals  between  key 
presses  is  their  variation  in  the  aided  mode  (Figure  5).  Sample  results  from  individual 
operators  using  aided  navigation  or  capitalizing  on  the  robot’s  initiative  to  follow  a  path 
defined  by  waypoints  and  to  avoid  obstacles  followed  different  strategies.  At  one  extreme 
was  the  monitoring  strategy  of  nibbling  at  the  course  (Figure  6,  Operator  A)  with 
planning  gaps  in  between  a  sequence  of  nibbles.  This  represented  a  small  difference  in 
performance  from  the  unaided  mode  on  this  metric.  At  the  other  extreme  was 
parsimony — long  planning  intervals  with  few  navigation  corrections  and  a  higher  risk  of 
making  errors  (Operator  D).  For  this  operator,  in  the  unaided  mode,  90%  of  the 
distribution  of  time  between  key  presses  was  5  seconds  or  less,  while  in  the  unaided 
mode  only  20%  was  in  this  range.  Intermediate  strategies  in  which  nibbling  and 
parsimony  were  alternated  also  occurred  (Operators  B  and  C).  In  all  cases,  aided 
navigation  resulted  in  more  “free  time”  than  unaided  navigation.  Why  one  strategy  was 
adopted  over  another  is  not  known,  but  reasons  could  include  trust  or  lack  thereof  in 
autonomy  or  simply  a  desire  to  exert  more  control  in  the  situation.  In  addition  to 
following  a  path,  operators  had  to  monitor  for  2D  obstacles,  unseen  by  the  robot,  and 
respond  to  navigation  problems  beyond  the  robot’s  capabilities.  The  errors  made  by 
crossing  or  overlapping  2D  obstacles  were  slightly  higher  for  aided  navigation, 
suggesting  that  operators  were  less  vigilant  in  this  mode.  Such  errors  were  the  one 
negative  attribute  of  using  aided  navigation. 

Overall,  aided  and  unaided  navigation  represent  a  continuum  of  robot-operator 
teamwork  interaction  that  ranged  from  little  to  no  interaction  (autonomous)  to  constant 
interaction  (teleoperation).  Unaided  robot  navigation  is  a  full-time  job  that  places  high 
demands  on  operator  attention;  aided  robot  navigation  demands  less  constant  attention 
from  the  operator.  Less  operator  attention  and  interaction  with  the  robot  generally  seems 
to  equate  to  more  errors,  suggesting  that  operator  vigilance  needs  emphasis  or  that  the 
navigation  technology  needs  enhancement  for  improved  performance. 

Operators  said  that  unaided  navigation  gives  the  operator  the  sense  of  being  in 
control.  In  contrast,  aided  navigation  was  said  to  be  good  for  “open  spaces”  but  not  for 
detailed  control  situations.  Although  operators  could  make  the  robot  move  identically  in 
either  mode,  that  is  not  what  they  did.  When  discriminating  robot  movements  were 
needed,  operators  dropped  out  of  aided  navigation  in  favor  of  unaided  control. 


18 


V.  CONCLUSION 


The  test  demonstrated  the  usefulness  of  evaluating  robot-operator  teams  to  assess 
success  and  failure  factors  in  the  performance  of  robots  and  the  effects  of  different 
technologies  on  that  performance.  The  results  of  measuring  robot-operator  perfonnance 
can  give  designers  and  program  managers  insight  into  the  factors  affecting  job 
performance  and  mission  success.  The  initial  results  reported  here  may  not  generalize 
across  different  robot-operator  teams  and  tasks;  however,  the  measurement  approach 
should  generalize  across  platfonns  and  technologies. 
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VI.  RECOMMENDATIONS 


The  measurement  concept  developed  and  tested  as  a  proof  of  principle  would 
benefit  from  additional  assessment  and  enhancements  before  transition  to  the  user 
community.  The  current  work  with  one  type  of  robot  and  limited  data  needs  expansion  to 
show  that  the  concept  generalizes.  The  next  steps  should  be  to  extend  the  measurement 
approach  to  more  complex  robot-operator  tasks  with  the  goal  of  providing  quantified 
answers  to  which  technologies  will  help  robot-operator  teams  better  accomplish  their 
jobs.  Additional  tests  should  include  data  logs  and  metrics  for  other  ground  robots  and  for 
air  robots.  To  make  implementation  easy,  the  schema  and  instrumentation  for  data 
collection  need  to  be  relatively  compatible  with  different  onboard  computers,  data¬ 
logging  systems,  and  user  interfaces  and  control  stations  of  various  robots.  The  most 
effective  way  for  that  to  happen  is  for  contractors  and  developers  to  integrate  the 
evaluation  methodology  into  their  robot  systems  and  planning  for  employment  under 
more  realistic  conditions  than  in  the  current  study. 
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APPENDIX  A— INSTRUCTIONS  FOR  PARTICIPANTS  IN  PROOF- 

OF-PRINCIPLE  TESTS 


A.  Overview 

Today,  you’ll  help  us  understand  how  to  measure  human-robot  teamwork.  The 
first  task  is  learning  and  practicing  how  to  operate  the  robot.  Then,  there  will  be  a  series 
of  tests  where  you  navigate  the  robot  through  a  course,  followed  by  searching  for  a 
simulated  bomb.  The  course  is  level  terrain  in  a  garage  with  boxes  and  floor  markings  to 
simulate  obstacles.  Sometimes  the  tests  are  with  navigation  assistance  from  the  robot 
(aided  mode)  and  sometimes  with  only  the  robot’s  sensors  (unaided  mode). 

You  can  see  the  robot  while  learning  how  to  operate  it.  But  for  the  tests,  you’ll  see 
only  a  control  display.  So,  practice  using  the  control  display  as  much  as  possible,  even 
during  learning. 

B.  Purpose 

The  purpose  of  the  tests  is  to  find  out  if  we  can  measure  how  well  you  and  a  robot 
perform  as  a  team.  You  give  commands  from  a  computer  terminal  that  causes  the  robot  to 
perform  navigation,  search,  and  find  tasks. 

1.  Learning  and  Practice 

Meet  the  Robot.  Ask  the  Observer  for  an  introduction  to  the  robot’s  features.  The 
start  point  for  training  is  in  the  area  you  see  in  front  of  the  control  station. 

2,  Unaided  Navigation  Using  Robot’s  Sensors 

Control  Display.  Look  at  the  robot’s  control  display  and  its  four  information 
windows: 

(1)  map  (left)  including  robot  position  and  activity 

(2)  feedback  window  (below  the  map) 

(3)  live  camera  video  (top  right) 

(4)  command  list  (lower  right) 
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Do  the  following  to  learn  about  the  control  display. 

1 .  Control.  Put  the  cursor  on  the  map  window  and  left  click  the  mouse. 

2.  Camera.  Zoom  in  with  the  robot’s  color  TV  camera  (D  key)  and  then  zoom 
out  again  (A  key).  Separately  press  each  key  a  few  times  until  you  end  up  at 
the  position  where  the  display  no  longer  changes.  Next,  tilt  the  camera  up  (W 
key)  and  then  look  down  (S  key).  Return  it  to  looking  ahead. 

3.  Turns.  Use  the  mouse  cursor  to  highlight  and  click  on  the  MOTORS  button 
(left  screen)  to  start  the  robot’s  motors.  Next,  left  click  on  the  map  window. 

a.  Locate  the  robot  symbol  (black  octagon  stop-sign  shape  on  the  map  and 
notice  the  extra  black  line  marking  the  front  of  the  robot. 

b.  Turn  to  the  left  by  pressing  the  left  arrow  key  three  times.  Each  key 
press  is  about  10  degrees  counterclockwise.  The  robot  is  a  bit  slow  to 
respond,  but  notice  the  red  rectangle  and  purple  arrow  combination  as 
the  robot  turns.  The  red  rectangle  shows  the  robot’s  expected  heading 
and  the  purple  arrowhead  its  current  heading.  The  long  green  arrow  is 
the  current  estimate  of  where  the  robot  is  pointed;  it  updates  slowly. 

c.  Turn  back  toward  the  right  (right  arrow  key)  using  two  key  presses.  Note 
that  pressing  the  left-left-left-right-right  commands  produce  a  net  10- 
degree  turn  counterclockwise. 

Stop  the  robot  (space  bar). 

Next,  turn  and  stop  the  robot  a  few  times  for  practice. 

4.  Forward- Backward  Movement. 

a.  Move  the  robot  forward  (up  arrow)  and  then  stop  (space  bar). 

b.  Move  the  robot  forward  and  look  at  the  map  display  to  see  the 
movement. 

c.  Stop  the  robot. 

d.  Press  the  up  arrow  key  twice  to  make  the  robot  move  at  its  highest 
speed. 

e.  Slow  the  robot  (down  arrow)  with  one  key  press  and  stop  with  two  key 
presses. 

f.  Press  the  down  arrow  key  again  to  put  the  robot  in  reverse  and  then  stop. 

Note:  The  up  arrow  tells  the  robot  to  go  forward,  and  there  are  two  forward 
speeds.  The  down  arrow  tells  the  robot  to  reverse  its  direction,  and  there  is  only  one 
speed.  You  can  slow  or  stop  the  robot  whether  going  forward  or  in  reverse  by  using  the 
arrows.  The  space  bar  stops  the  robot  no  matter  what  it’s  doing. 
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5.  Maneuver. 


a.  Drive  the  robot  toward  the  green  circle  marked  as  “Gate  I”  on  the  map. 
A  cardboard  box  on  either  side  of  the  green  circle  makes  the  gate. 

b.  Bump  a  box  on  one  side  of  Gate  1  with  the  robot  so  the  robot  crashes 
and  stops. 

c.  Look  at  the  robot  symbol  on  the  map  and  the  message  in  the  feedback 
window  (below  the  map)  showing  the  robot  is  stopped  and  stuck. 

d.  To  continue,  back  up  the  robot.  Then,  stop  either  with  the  up  arrow  key 
or  the  space  bar. 

e.  Notice  the  green  dots  (laser  range  finder)  and  blue  dots  (sonar 
reflections)  that  outline  what  objects  these  sensors  are  detecting.  Clear 
space  between  the  robot  and  a  dot  means  that  no  object  is  there.  Sonar 
sees  close  to  the  robot’s  back  and  front.  The  laser  sees  farther  forward 
but  not  behind.  The  black  squares  show  known  and  established  walls 
and  other  objects.  Note  that  you’ll  come  upon  unmapped  objects,  too 
(that  can  only  be  seen  with  the  robot’s  camera). 

f.  Next,  maneuver  the  robot  and  go  through  Gate  I  and  stop. 

g.  Continue  by  maneuvering  the  robot  through  Gates  II,  III,  and  IV — but 
avoid  hitting  boxes  at  the  gates,  boxes  that  are  obstacles  on  the  course, 
and  2D  black-and-yellow  squares  on  the  floor  (that  you  can  find  only 
with  the  camera). 

h.  Practice  until  you  are  comfortable  with  how  the  robot  turns  and  responds 
to  your  commands  and  how  the  display  shows  what’s  going  on.  Make 
sure  to  practice  around  gates  and  obstacles  to  learn  how  best  to  turn  the 
robot. 

3.  Aided  Navigation  Using  Waypoints 

In  the  aided  test,  the  robot  helps  you  navigate  a  route  using  points  along  the  way 
you  want  to  go  (waypoints)  that  you  select  with  the  mouse  cursor.  Waypoints  are  a 
feature  in  addition  to  all  other  control  features. 

Do  the  following  to  learn  about  using  waypoints, 

1.  Start  Waypoint  Mode.  Turn  on  BEHAVIORS  (left  button  mouse  click)  to 
start  the  waypoint  mode.  (The  marker  in  the  left  display  window  changes 
from  gray  to  yellow,  and  the  robot  now  will  move  before,  during,  or  after  you 
select  one  or  more  waypoints.)  After  clicking  on  BEHAVIORS,  you  must 
left  mouse  click  on  the  map  so  that  the  robot  will  accept  the  command. 
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2.  Mark  a  Waypoint.  Decide  on  a  waypoint  and  mark  it  with  the  mouse  cursor 
(Ctrl  +  left  mouse  button).  Notice  the  green  square  on  your  display  that 
marks  that  waypoint. 

3.  Maneuver  with  Waypoints.  Next,  mark  many  waypoints  to  navigate  through 
the  gates  without  hitting  3D  obstacles  (boxes)  and  without  crossing  2D 
obstacles  (yellow-striped  squares).  Notice  that  red  dots  mark  waypoints  after 
the  current  (green)  one  on  your  display. 

4.  Change  waypoints.  Erase  a  waypoint  (backspace)  and  then  plot  new 
waypoints.  That’s  one  way  to  change  a  route. 

5.  Insert  New  Waypoints.  Use  the  mouse  cursor  to  insert  a  waypoint  in  between 
those  already  (Alt  +  left  mouse  button).  Notice  how  this  insert  command 
differs  from  what  you  did  earlier  to  mark  one  waypoint  after  another.  That’s 
another  way  to  change  a  route. 

6.  Waypoint  Errors.  Force  an  error  by  placing  a  waypoint  so  the  robot  will  hit  a 
box. 

a.  After  the  robot  stops,  turn  off  BEHAVIORS  on  the  display  (left  button 
mouse  click)  so  you’re  in  non-aided  mode.  Manually  move  the  robot 
away  from  the  box. 

b.  Then,  enter  waypoints  with  the  mouse,  turning  on  BEHAVIORS 
whenever  you’re  ready  for  robot  movement. 

7.  Practice  adding  waypoints  (and  editing  them)  by  navigating  through  Gates  I, 
II,  III,  and  IV  and  avoiding  obstacles  around  the  training  area.  Continue  until 
comfortable. 

4.  Failed  Waypoint  Navigation 

8.  Waypoint  Confusion.  Sometimes  the  robot  can’t  find  a  path  to  the  next 
waypoint  (even  if  there  is  one). 

a.  Ask  the  Observer  to  put  an  obstacle  on  the  training  course  to  cause  the 
problem. 

b.  Select  the  new  waypoint  and  observe  what  happens.  Also,  notice  the 
message  on  your  feedback  window. 

Note:  The  waypoint  planner  fails  when  it  doesn’t  sense  a  large  enough  space  for 
the  robot  to  navigate.  Turn  off  BEHAVIORS  mode  and  navigate  manually  to  a  different 
location.  Turn  on  BEHAVIORS  again  and  try  new  waypoints. 
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5.  Begin  Testing  Session 

The  test  is  to  navigate. 

1 .  Testing  Start  Point.  Ask  the  Observer  to  move  the  robot  to  the  testing  start 
point  and  reset  the  computer  while  you  take  a  short  break. 

2.  Navigate  Task.  The  navigate  task  is  to  drive  the  robot  through  a  series  of 
numbered  gates  either  aided  with  waypoint  navigation  or  unaided  using  the 
robot’s  camera  and  sensors.  The  course  has  several  possible  numbered  paths. 
Notice  how  a  hard  copy  of  the  map  has  navigation  reference  points  marked 
by  green  circles  and  roman  numerals.  For  each  test,  you  pass  the  reference 
points  in  the  order  shown  and  go  between  gates  (pairs  of  numbered 
boxes/obstacles)  to  complete  the  course. 

Signal  your  find  (B  key  for  bomb)  on  the  computer  and  tell  the  Observer 
when  you’re  done. 

3.  Tasks.  The  Observer  will  tell  you  which  gates  to  go  through,  either  aided  or 
unaided,  after  you’re  done  reading  these  instructions.  Note  that  in  the  aided 
mode,  all  controls  for  unaided  operation  still  work.  Your  navigation  task  is  to 
move  as  fast  as  you  can  while  avoiding  3D  obstacles  (boxes)  and  2D 
obstacles  marked  on  the  floor  by  yellow-and-black  squares.  Some  obstacles 
appear  on  the  map  but  there  also  are  unmapped  obstacles.  There’s  a 
turnaround  point  marked  by  two  stacked  boxes  at  the  far  end  of  the  course. 

4.  Reminders.  Get  your  first  test  course  assignment  and  make  sure  you  know  if 
the  task  is  aided  (waypoint  navigation)  or  unaided  (sensor  navigation).  Use 
the  robot’s  sensors  to  best  advantage  to  avoid  known  and  unknown,  2D  and 
3D  obstacles.  Speed  is  urgent  in  finding  the  mock  bomb! 

Use  waypoint  commands  whenever  available. 

5.  Questions?  When  you’re  ready,  proceed  to  the  next  page  for  course 
navigation  information. 
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APPENDIX  B— TEST  PATHS  AND  TASKS 


A.  Navigate  the  Robot  Course  Using  the  Following  Order 


A  IV  V  XIV  VIII  III 


(up  -  unaided) 


B  I  VIII  XIV  V  IV 


(back  -  aided) 


E  IV  VII  X  IX  XI  III  (up -unaided) 


D  I  VIII  IX  VI  V  IV  (back  -  aided) 


C  XII  VII  VI  VIII  XI  III  (up  -  aided) 


F  I  VIII  XIII  X  II  IV  (back -unaided) 
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y  Practice  runs 
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